APPLICATION NO. 



09/783,725 



FILING DATE 



02/14/2001 



FIRST NAMED INVENTOR 



Thomas R. Firman 



ATTORNEY DOCKET NO. CONFIRMATION NO. 



29973 7590 08/18/2009 

CAREY, RODRIGUEZ, GREENBERG & PAUL LLP 
ATTN: STEVEN M. GREENBERG, ESQ. 
950 PENINSULA CORPORATE CIRCLE 
SUITE 3020 

BOCA RATON, FL 33487 



ARMSTRONG, ANGELA A 



PAPER NUMBER 



DELIVERY MODE 



Please find below and/or attached an Office communication concerning this application or proceeding. 

The time period for reply, if any, is set in the attached communication. 



PTOL-90A (Rev. 04/07) 



l/ffflrC? nVrliUli Otfff Iff ids y 


Application No. 

09/783,725 


Applicant(s) 

FIRMAN, THOMAS R. 


Examiner 

ANGELA A. ARMSTRONG 


Art Unit 

2626 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address — 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )^| Responsive to communication(s) filed on 26 May 2009 . 
2a )£3 This action is FINAL. 2b)^ This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) ^3 Claim(s) 4-7, 14,16-57 and 60-62 is/are pending in the application. 

4a) Of the above claim(s) 4,5 and 16-34 is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) |EI Claim(s) 6-7,14,35-57,60-62 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) IZI The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)^ accepted or b)^ objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12)D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 0 Certified copies of the priority documents have been received. 

20 Certified copies of the priority documents have been received in Application No. . 

3.Q Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) □ Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Paper No(s)/Mail Date. . 

3) □ Information Disclosure Statement(s) (PTO/SB/08) 5 ) □ Notice of Informal Patent Application 

Paper No(s)/Mail Date . 6) □ Other: . 



PTOL-326 (Rev. 08-06) Office Action Summary Part of Paper No./Mail Date 



Application/Control Number: 09/783,725 Page 2 

Art Unit: 2626 

DETAILED ACTION 

1 . This Office Action is in response to the Remarks filed May 26, 2009. Currently, claims 
4-7, 14, 16-57, and 60-62 are pending, with claims 4-5 and 16-34 being withdrawn from 
consideration. 

Claim Rejections - 35 USC §103 

The text of those sections of Title 35, U.S. Code not included in this action can be found 
in a prior Office action. 

2. Claims 14 and 60-62 are rejected under 35 U.S. C. 103(a) as being unpatentable over 
Goldhor (US Patent No. 5,231,670) in view of Porter (US Patent 4,829,576). 

3. Regarding claims 14 and 60-62, Goldhor teaches a system and or method for generating 
text from a voice input that divides the processing of each speech event into a dictation event 
and a text event. The teachings of Goldhor provide for the system and method to process both 
simple spoken words as well as spoken commands and to provide the necessary text generation 
in response to the spoken words or to execute an appropriate function in response to a 
command. Speech recognition includes the ability to distinguish between dictation text and 
commands. (Figure 1, elements 12, 14, 16, 18, "recognizer"; col. 1, lines 17-20 55-68; col. 2, 
lines 1-2; col. 4, lines 10-13; col. 5, lines 40-55; col. 6, lines 46-48). Which reads on a voice 
user interface device comprising means for converting a voiced utterance into a corresponding 
signal as an input to a computer or into an internal command to the interface device and means 
for recognizing the voiced utterance as either one to be converted to said signal or as one to be 
converted to said command, since the system specifically receives the input speech and 
determines if the speech is for text or is a command and makes changes to new sets of 
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utterances when creating new dictation events or new text events. Goldhor does not provide for 
the converter to determine the changing commands. Porter discloses a voice recognition system 
for providing speech recognition systems, which recognize commands for use with a text editor. 
The recognition system is used with command words for finding files, searching for text strings, 
implementing a scheme for selecting menu choices, such that menus are manipulated so as to 
display a particular of choices (search or find file) and for selection of the desired displayed 
command (Col. 7, line 33 to col. 8, line 31; col. 18, line 44 to col. 21, line 5; Figure 41; col. 21, 
line 39 to col. 22, line 34). Porter specifically teaches the system is designed so as to improve 
the ease and reliability with which humans can control computer systems which deal with 
elements contained in data structures, such as words in text files or names in a database (col. 2, 
lines 15-27). It would have been obvious to one of ordinary skill at the time of the invention to 
modify the system of Goldhor to implement changing a set of voiced utterances or commands to 
be recognized based on an input command as taught by Porter, for the purpose of improving the 
ease and reliability of the recognizer, as also suggested by Porter. 

4. Claims 6-7 are rejected under 35 U.S.C. 103(a) as being unpatentable over Goldhor (US 
Patent No. 5,231,670) in view of Torres (US Patent No. 4,821,211). 

5. Regarding claim 6, Goldhor teaches a voice user interface system for producing input to a 
computer, and a program for execution on said computer, a state of said program, said 
configuration being associated with control of said program, comprising a voice recognizer for 
recognizing a voiced utterance and for providing corresponding signals as input to said computer 
(Figure 1, elements 12, 14, 16, 18, "recognizer"; col. 1, lines 17-20 55-68; col. 2, lines 1-2; col. 
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4, lines 10-13; col. 6, lines 46-48), and a converter for converting said voiced utterance into a 
command string (col. 1, lines 17-20 55-68; col. 2, lines 1-2; col. 4, lines 10-13; col. 6, lines 46- 
48). 

Goldhor does not specifically teach mimicking mouse commands or various details 
related to the display of the graphical user interface. Torres teaches a method and apparatus for 
Navigating among program menus using a graphical menu tree and provides a description of 
well known cursor functionality and graphical user interface display for permitting computer 
users to access computer applications and manage windows by graphically designating graphic 
representations (which reads on "graphical elements") and manipulating those graphical 
representations ("graphical elements") via a graphic pointing device or voice interaction (Figures 
1-3 and 4D; col. 3, line 64 to col. 5, line 56). 

It would have been obvious to one of ordinary skill at the time of the invention to provide 
for the manipulation of the movement of the cursor via vocal interaction as taught by Torres, in 
the system of Goldhor, because this would enable a person who is not physically able to move 
the cursor to be able to move the cursor or access and control icons/applications via vocal 
commands. 

Torres does not specifically teach the manipulation of graphical representations via 
specifying coordinates. However, manipulating graphical representations and facilitating 
movement on a graphical display by specifying coordinates of desired locations was well known. 
It would have been obvious to one of ordinary skill at the time of the invention to provide for the 
manipulation of the movement of the cursor via vocal interaction as taught by Torres, and to 
allow for vocal command movements via specified coordinates, as was well known, in the 
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system of Goldhor, for the purpose of enabling a person who is not physically able to move the 
cursor to be able to move the cursor or access and control icons/applications via vocal 
commands. 

6. Regarding claim 7, Goldhor teaches command string further comprises a command to 
said program at (col. 1, lines 17-20 55-68; col. 2, lines 1-2; col. 4, lines 10-13; col. 6, lines 46- 
48). 



8. Claims 35-40 and 52-57 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Torres (US Patent No. 4,821,211). 

9. Regarding claim 35, Torres teaches a method and apparatus for navigating among 
program menus using a graphical menu tree and provides a description of well known cursor 
functionality and graphical user interface display for permitting computer users to access 
computer applications and manage windows by graphically designating graphic representations 
and manipulating those graphical representations via a graphic pointing device or voice 
interaction (Figures 1-3 and 4D; col. 3, line 64 to col. 5, line 56), which reads on a method for 
use with a machine having a graphical user interface and an application program, the method 
comprising the graphical user interface being controlled at least in part by a control signal that 
can be invoked in response to the pointing device and the graphical user interface enabling a user 
to launch the application program and receiving a voice utterance from a user. 

Torres does not specifically disclose launching the application program in response to the 
received voiced utterance without invoking the control signal. However, the teachings of Torres 
specifically disclose using voice interaction to control system functionality (col. 4, lines 16-17). 
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It would have been obvious to one of ordinary skill at the time of the invention to use the 
system of Torres to launch application programs via voice control and interaction so as to 
provide computer access to application programs for physically challenged individuals. 

Regarding claim 36, Torres teaches an operating system with a graphical interface at (col. 

4, lines 3-6). 

Regarding claim 37, Torres teaches the graphical user interface is shown on a display at 
col. 3, line 67. 

Regarding claim 38, Torres teaches the machine comprises a computer (col. 4, lines 1-7). 
Regarding claim 39, similar limitations to claim 35 are discussed above. Additionally 
Torres teaches there are multiple application programs (Figures 1-3 and 4D; col. 3, line 64 to col. 

5, line 56). 

Regarding claim 40, similar limitations to claim 35 are discussed above. Additionally, 
Torres teaches there are multiple application programs, an operating system provides the 
graphical user interface, the graphical user interface is shown on a display, and the machine 
comprises a computer (Figures 1-3 and 4D; col. 3, line 64 to col. 5, line 56), 

Regarding claim 52, similar limitations to claims 35, 41 and/or 45 are discussed above. 
Additionally, Torres teaches selectable menu items (Figures 1-3 and 4D). 

Regarding claim 53, similar limitations to claim 35 are discussed above. Additionally 
Torres teaches there are multiple graphical representations (Figures 1-3 and 4D; col. 3, line 64 to 
col. 5, line 56). 

Regarding claim 54, Torres teaches the graphical user interface is shown on a display at 
col. 3, line 67. 
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Regarding claim 55, Torres teaches the machine comprises a computer (col. 4, lines 1-7). 

Regarding claim 56, Torres does not specifically disclose performing a function 
associated with a menu item in response to a second received voiced utterance. However, the 
teachings of Torres describe the well-known implementation of a pointer for accessing computer 
applications and disclose using voice interaction to control system functionality. 

It would have been obvious to one of ordinary skill at the time of the invention to use the 
system of Torres to provide application programs or other menu item functionality via second or 
additional received voiced utterances so as to provide computer access to application programs 
for physically challenged individuals. 

Regarding claims 57, Torres teaches manipulating other graphical items includes altering 
size and location of a window (Figures 1-3 and 4D; col. 3, line 64 to col. 5, line 56). 

10. Claims 41-51 are rejected under 35 U.S.C. 103(a) as being unpatentable over Torres (US 
Patent No. 4,821,21 1) in view of Porter (US Patent No. 4,829,576). 

1 1 . Regarding claim 4 1 , Torres teaches a method and apparatus for navigating among 
program menus using a graphical menu tree and provides a description of well known cursor 
functionality and graphical user interface display for permitting computer users to access 
computer applications and manage windows by graphically designating graphic representations 
and manipulating those graphical representations via a graphic pointing device or voice 
interaction (Figures 1-3 and 4D; col. 3, line 64 to col. 5, line 56), which reads on a method for 
use with a machine having a graphical user interface and an application program, the method 
comprising the graphical user interface being controlled at least in part by a control signal that 
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can be invoked in response to the pointing device. Torres does not teach manipulating a 
graphical item separately from the cursor. Porter discloses a voice recognition system for 
providing speech recognition systems, which recognize commands for use with a text editor. 
The recognition system is used with command words for finding files, searching for text strings, 
manipulating menus so as to display a particular menu of choices (search or find file) and for the 
selection of the desired displayed command (Col. 7, line 33 to col. 8, line 31; col. 18, line 44 to 
col. 21, line 5; Figures 36-37 and 40-41; col. 21, line 39 to col. 22, line 34). Porter specifically 
teaches the system is designed so as to improve the ease and reliability with which humans can 
control computer systems which deal with elements contained in data structures, such as words 
in text files or names in a database (col. 2, lines 15-27). It would have been obvious to one of 
ordinary skill at the time of the invention to modify the system of Torres to implement changing 
a set of voiced utterances to be recognized based on an input command as taught by Porter, for 
the purpose of improving the ease and reliability of the recognizer, as also suggested by Porter. 

Regarding claim 42, Torres teaches an operating system with a graphical interface at 
(col. 4, lines 1-17). 

Regarding claim 43, Torres teaches the graphical user interface is shown on a display at 
col. 3, line 67. 

Regarding claim 44, Torres teaches the machine comprises a computer (col. 4, lines 1-7). 

Regarding claim 45, similar limitations to claim 41 are discussed above. Additionally 
Torres teaches there are multiple graphical items and manipulating the other graphical items 
(Figures 1-3 and 4D; col. 3, line 64 to col. 5, line 56), as various menu items, windows or 
graphical representations to which the user has access and control. 



Application/Control Number: 09/783,725 Page 9 

Art Unit: 2626 

Regarding claims 46-5 1 , Torres teaches manipulating other graphical items includes 
altering size and location of a window (Figures 1-3 and 4D; col. 3, line 64 to col. 5, line 56). 

Response to Arguments 

Applicant's arguments filed May 26, 2009, have been fully considered but they are not 
persuasive. Regarding claims 14 and 60-62, Applicant traverses the Examiner's position that 
the claimed invention would be obvious in view of the combination of the teachings of Goldhor 
and Porter. Applicant has argued that Goldhor does not disclose a voice utterance is recognized 
either as an utterance to be converted to a corresponding signal or as a voice utterance to be 
converted to a command to a program. In response, the Examiner argues that Goldhor provides 
support for the claimed limitation since, at col. 3, line 38 to col. 4, line 57, Goldhor teaches the 
speech event analyzer 16 generates a list or set of possible candidates that represent the voice 
input processed by the speech signal processor 14. The speech event analyzer 16 transmits the 
candidate sets to a dictation event subsystem 18. The dictation event subsystem 18 analyzes the 
candidate sets and chooses the "BEST MATCH", i.e. the candidate with the highest degree of 
similarity. This candidate is then considered the correct translation, and the dictation event 
subsystem forwards the translation to text event subsystem 20 which in turn inputs the 
translated text to an application. Additionally, Goldhor teaches that when a particular candidate 
is chosen as best match generally represents the spelling of that particular candidate word or 
phrase and that the translation may also be any other legal input into a particular application, 
and the translation may in fact be used to control the application by voice. The translation may 
also include input to the recognizer whereby the operation of the recognizer can be controlled 
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and its state changed. Thus, Goldhor's translation which may be the input as translated text 
into a particular application or which may be used to control (command) the application by 
voice provides support for a voice utterance is recognized either as an utterance to be converted 
to a corresponding signal or as a voice utterance to be converted to a command to a program. 

Regarding claims 6 and 7, Applicant traverses the Examiner's position that the claimed 
invention would be obvious in view of the teachings of Goldhor and Torres. Applicant has 
argued Goldhor neither teaches nor contemplates a "command string" that includes a command 
positioning a pointer "at coordinates specified relative to a graphical element of a configuration 
other than the pointer." In response to applicant's arguments against the references individually, 
one cannot show nonobviousness by attacking references individually where the rejections are 
based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 
1981); In reMerck& Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). As indicated in the 
previous rejection and in the rejection above, Goldhor was cited as teaching a voice user 
interface system for producing input to a computer, and a program for execution on said 
computer, a state of said program, said configuration being associated with control of said 
program, comprising a voice recognizer for recognizing a voiced utterance and for providing 
corresponding signals as input to said computer, and a converter for converting said voiced 
utterance into a command string. Since Goldhor does not specifically teach mimicking mouse 
commands or various details related to the display of the graphical user interface, Torres was 
cited as teaching a method and apparatus for Navigating among program menus using a 
graphical menu tree and provides a description of well known cursor functionality and graphical 
user interface display for permitting computer users to access computer applications and manage 
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windows by graphically designating graphic representations ( "graphical elements") and 
manipulating those graphical representations ("graphical elements") via a graphic pointing 
device or voice interaction. Since Torres does not specifically teach the manipulation of 
graphical representations via specifying coordinates and since manipulating graphical 
representations and facilitating movement on a graphical display by specifying coordinates of 
desired locations was well known, one of ordinary skill would have recognized the advantages of 
modifying the system to provide for the manipulation of the movement of the cursor via vocal 
interaction as taught by Torres, and to allow for vocal command movements via specified 
coordinates, as was well known, in the system of Goldhor, since such a modification would have 
yielded predictable results and resulted in an improved system by enabling a person who is not 
physically able to move the cursor to be able to move the cursor or access and control 
icons/applications via vocal commands. 

Regarding claims 35-40 and 52-57, Applicant traverses the Examiner's position that the 
claimed invention would be obvious in view of the teachings of Torres. Applicant has argued 
Torres does not disclose or render obvious launching an application program without invoking a 
control signal that can be invoked in response to a pointing device on a machine. Applicant also 
argues Torres does not disclose or render obvious displaying a menu without invoking a control 
signal that may be sent by a pointing device. In response, the Examiner argues Torres teaches a 
method and apparatus for navigating among program menus using a graphical menu tree and 
provides a description of well known cursor functionality and graphical user interface display for 
permitting computer users to access computer applications (launch computer application 
programs) and manage windows by graphically designating graphic representations and 



Application/Control Number: 09/783,725 Page 12 

Art Unit: 2626 

manipulating those graphical representations via voice interaction. Torres suggests control is 
provided by either keyboard, mouse, touch screen or voice interaction, so as to provide control 
by keyboard or mouse or touch screen or voice interaction. Torres' specific teaching that the 
system allows for navigating program menus and the well known cursor functionality for 
permitting users to access computer applications and manage windows which can be achieved 
via voice interaction provides adequate support for launching an application program without 
invoking a control signal that can be invoked in response to a pointing device on a machine as 
well as displaying a menu without invoking a control signal that may be sent by a pointing 
device, since he specifically disclose using voice interaction to control system functionality (col. 
4, lines 16-17). 

Regarding claims 41-51, Applicant traverses the Examiner's position that the claimed 
invention would be obvious in view of the teachings of Torres in view of Porter. Applicant has 
argued Porter teaches the manipulation of textual items not a "one other graphical item" within a 
graphical user interface. In response to Applicant's arguments, the Examiner argues since Torres 
was cited for teaching manipulation of graphical representations, and Porter's system clearly 
provides for manipulating items displayed to the user via voice commands as well as 
manipulating menus so as to display a particular menu of choices (search or find file) and for the 
selection of the desired displayed command without the cursor, the combination Torres and 
Porter provide adequate support for the manipulation of graphical items separately from the 
cursor, as indicated in the previous rejection and repeated in the rejection above. 
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Conclusion 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to ANGELA A. ARMSTRONG whose telephone number is 
(571)272-7598. The examiner can normally be reached on Monday-Thursday 1 1 :30-8:00 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Richemond Dorvil can be reached on 571-272-7602. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 



Application/Control Number: 09/783,725 Page 14 

Art Unit: 2626 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



/Angela A Armstrong/ 

Primary Examiner, Art Unit 2626 



